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Abstract 

We present the result of a search for a massive color-octet vector particle, (e.g. a 
massive gluon) decaying to a pair of top quarks in proton-antiproton collisions 
with a center-of-mass energy of 1.96 TeV. This search is based on 1.9 fb _1 
of data collected using the CDF detector during Run II of the Tevatron at 
Fcrmilab. We study tt events in the lepton+jets channel with at least one b- 
taggcd jet. A massive gluon is characterized by its mass, decay width, and the 
strength of its coupling to quarks. These parameters are determined according 
to the observed invariant mass distribution of top quark pairs. We set limits on 
the massive gluon coupling strength for masses between 400 and 800 GeV/c 2 
and width-to-mass ratios between 0.05 and 0.50. The coupling strength of the 
hypothetical massive gluon to quarks is consistent with zero within the explored 
parameter space. 

Keywords: massive gluon, top quark 
PACS: 13.85.Rm, 14.65.Ha, 14.80.-j 



1. Introduction 



The top quark is the heaviest known elementary particle, with a mass very 
close to the electroweak symmetry-breaking scale. As such, the top could be sen- 
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sitive to physics beyond the standard model (SM) New particles decaying 

to ti pairs can be scalar or vector, color-singlet or color-octet; a scalar resonance 
is predicted in two-Higgs-doublets models @, 0| jvector particles appear as mas- 
sive Z-like bosons in extended gauge theories [5|], or as Kaluza-Klein states of 
the gluon and Z boson [H, 0, or as colorons Q . Searches for a color-singlet par- 
ticle decaying to a ti pair have been performed by CDF and DO collaborations 
in Run I @, Q and Run II [HI [LI [ljj. 

In this Letter we describe a search for a massive color-octet vector particle G, 
which we call generically a "massive gluon" . We assume G has a much stronger 
coupling to the top quark than to the lighter quarks, q = (u, d, c, s, b) [lj]. We also 
assume the massive-massless gluon coupling is negligible. A Feynman diagram 
for ti production via massive-gluon is shown in Fig. [T] The coupling strength 
of massive gluons to light and top quarks are X q g s and Xtg s , respectively, where 
g s is the coupling constant of the SM strong interaction. 

In tt production, there are three observable parameters: the product of 
massive gluon coupling strength A = X q X t , mass M, and width T. In this 
analysis we consider only the possibility of novel strong-sector production; we 
assume that the weak decay of the top quark follows the SM. Since the color 
and the current structures of G and SM gluon (<?) are identical, interference 
between processes through massive gluon and massless gluon will produce a ti 
invariant mass distribution with an enhanced signal that has a characteristic 
form [l[ as shown later in Fig. [31 If the coupling of G to quarks is assumed to 
be parity-conserving, the production matrix element can be written as 



|AVod.| 2 - ^s 2 (2-/3 2 +/3 2 cos 2 0)(n 3 + An int . +A 2 n G ), (1) 

where s, (3, and 9 are the ti invariant mass squared, the velocity of the top 
quark, and the angle between the top quark and the incident quark in the ti 
center of mass system, respectively. The propagator factors of gluons, massive 
gluons, and interference are 



, 1 n - - S - M2 

B-p' G - (J _ Af 2)2 + M 2 T 2 ' int - ~ I(s-M 2 ) 2 +M 2 r 2 ' 



We search for ti pairs produced by massive and massless gluons, where inter- 
ference between these diagrams (denoted by g + G) is considered, by examining 
the invariant mass spectrum of observed ti candidate events. 
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Figure 1: A Feynman diagram showing ft production via a massive gluon in the "lepton+jets" 
decay channel. 



2. Selection of candidate events 

The search is based on data collected with the CDF II detector between 
March 2002 and May 2007 at the Fermilab Tevatron pp collider, corresponding 
to an integrated luminosity of about 1.9 fb _1 . The CDF II detector is a gen- 
eral purpose detector which is azimuthally and forward-backward symmetric. 
The detector consists of a charged particle tracking system composed of silicon 
microstrip detectors and a gas drift chamber inside a 1.4 T magnetic field, sur- 
rounded by electromagnetic and hadronic calorimeters and enclosed by muon 
detectors. The details of the detector are described elsewhere fl3l ] . 

The cross section for standard model tt production in pp collisions at y/s = 
1.96 TcV is dominated by qq annihilation (~ 85%). The remaining ~ 15% is 
attributed to gluon-gluon fusion [14| . Standard model top quarks decay almost 
exclusively to Wb. The search presented here focuses on the tt event topology 
wherein one W boson decays hadronically while the other decays to an electron 
or muon and its corresponding neutrino. Both b quarks and the two decay 
quarks of the second W boson appear as jets in the detector. Accordingly, tt 
candidates in this "lepton + jets" channel are characterized by a single lepton, 
missing transverse energy fix [l5l |. due to the undetected neutrino, and four 
jets. 

We use lepton triggers that require an electron or muon with pt > 18 GeV/c. 
Events included in our analysis must first contain an isolated electron (muon) 
with Et > 20 GeV (pr > 20 GeV/c) in the central detector region with 

< 1.0. Electron and muon identification methods are described in Ref. [l6j |. 
We remove events which have multiple leptons, cosmic ray muons, electrons 
from photon-conversion, or tracks, such that its momenta added to the primary 
lepton momenta gives an invariant mass equal to the Z mass. The position 
of the primary vertex (along the beam) is required to be within 60 cm of the 
center of the nominal beam intersection and consistent with the reconstructed 
z position of the high-pr lepton. Events must also feature at least 20 GeV 
of Pt attributable to the presence of a high-p^ neutrino, as well as exactly 
four jets with \rj\ < 2.0 and Et > 20 GeV (jet Et corrections are described 
in Ref. [l3|)- Jets are clustered with a cone-based algorithm, with a cone size 
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Table 1: Background composition to the qq — » tt process and the expected numbers of events. 
Systematic uncertainties coming from the background estimation method are listed. Elec- 
troweak includes single top quark, diboson production, and Z bosons + jets productions. A 
luminosity of 1.9 fb" 1 is assumed. 



Source 


Expected number of events 


Elcctroweak 


9.9 ±0.5 


W + bottom 


16.5 ±6.7 


W + charm 


12.9 ±5.2 


Mistags 


16.7 ±3.6 


non-W 


13.6 ± 11.7 


SM gg -> tt 


48.2 ± 15.6 


Total Background (nf*^ ± cr?*"^) 


117.8 ± 19.8 


SM qq -> tt (<r = 5.6 pb) 


211. 7± 29.3 


Data 


371 



AR = a/ A(p 2 + Arf — 0.4. We reduce non-ii backgrounds by requiring at least 
one jet identified by the displaced secondary vertex "6-tagging" algorithm [l]| 
as being consistent with the decay of a long-lived b hadron. 



3. Background 

The background to the tt signal is mostly from VF±jets events with a falsely- 
reconstructed secondary vertex (Mistags) or from W±jets events where one or 
more jets are due to heavy-flavor quarks. Smaller contributions are from QCD 
multi-jet production, in which either the W signature is faked when jets contain 
semi-leptonic 6-hadron decays or when jets are mis-reconstructed and appear 
as electrons and missing Et, single top quark production, diboson (WW, WZ, 
ZZ) production, and Z bosons produced in association with multiple jets. The 
methods used to estimate the backgrounds are detailed in Ref. [13[ . The gg — > tt 
process is taken as a background for this search, which is estimated by assuming 



0.16 ± 0.05 gluon fusion fraction from Ref. [14(. The estimated backgrounds in 
the sample are summarized in Table U\ The diboson and gg — ► tt background 
processes are modeled with pythia [18j , H / ±jets and Z+jets processes with 
ALPGEN [l9|], and QCD with data. 



4. tt invariant mass reconstruction 

We search for a massive gluon by examining the tt invariant mass (vlti) 
spectrum of the selected events. Our analysis procedure consists of two steps. 
(A) We first reconstruct the parton level momentum set of tt = (bl+y)(bl~D), 
event-by-event, with the Dynamical Likelihood Method (DLM) [2(1 El. The 
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likelihood in DLM is defined by the differential cross section per unit phase space 
volume, multiplied by a posterior transfer function (TF), which is a probability 
density function from the observed to the parton kinematic quantities. For a 
given event, different parton kinematics set are inferred with TF. We assume 
in this paper the TF is independent of the production matrix for it, hence in 
the reconstruction, we remove the it production matrix from the likelihood. For 
each event the reconstructed \/§ti 1S averaged over inferred parton paths, and 
denoted by \fs r . We denote the true V~s t t at the parton level by y/I p . 
(B) We interpret V§ r as an observed quantity and apply a standard unbinned 
likelihood technique to reproduce the V§ r distribution using Monte Carlo (MC) 
events, which consist of signal and relevant background processes. 

The V§ r distribution for the signal g + G process is expressed relative to 
[dc/dV^plsM-qg^ti' ^ ne cross section after the event selection cuts (ex- 
pressed by *) by 

* 

R(y§p',a)f(V§ r - V%; \fip)d\fi p , (3) 

SM.qq^tt 

where a. = (A, M,T) is a set of massive gluon parameters, and N(a) is the 
normalization factor. The ratio of g + G to SM it production cross sections is 

R <^°» = 1 + 2A (5^wfs^ + A2 (3^WTl^- < 4 > 

In this ratio, PDF effects, top propagators, decay matrix elements and the final 
state densities for g + G and SM processes cancel out, making it possible to 
generate (or derive) g + G events with specified a from standard model it MC 
events. The resolution function / translates y/l p to \fi r . It is obtained from 
the scatter plot of VJ r — V§ p vs. y/l p using the PYTHIA SM qq — * it MC 
sample, as shown in Fig. [2j The peak and half maximum points of — V§p 
are typically -0.2tH 7 r , -0.9±H ° 7 , and — 3.5±||:f GeV/c 2 at ^ = 400, 600, 
and 800 GeV/c 2 , respectively. 



N(a) 



da 

d i \/Ip 
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350 400 450 500 550 600 650 700 750 8/JO 
True tt invariant mass [GeV/c ] 



Figure 2: Resolution function f(y; x), where x = and y = V§r — V$p- 



Figure [3] shows an example of p s [VJ r ; a] . The first (second) peak is due 
to g(G). At the parton level, the G peak(dip) is above(below) M for posi- 
tive (negative) A, while at the reconstructed Vs r distribution the peak(dip) is 
shifted due to the resolution function. The pythia MC with M t = 175 GeV/c 2 
is used to estimate [da / d^plgM-qq^tr 




$00 300 400 500 600 700 800 900 1000 

Reconstructed tf invariant mass [GeV/c 2 ] 



Figure 3: Signal probability density function p s [VS r ; a] with M = 600 GeV/c 2 , Y/M = 0.10, 
and A = ±0.3. 



5. Extraction of coupling strength 

If M and T/M are given, the V§ r distribution of the data is a function of A 
and the numbers of the signal and background events, n s and rib. By applying 
a three parameter unbinned likelihood method for (\,n Sl rib), the likelihood 
function L for the distribution is 
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L \X,n s ,n b \M,— J = G(n b ;n b 1 ,a b ^)P(N;n) [[ - . 

Function G(n&; , o^ x ^) constrains the total background normalization with 
Gaussian probability around the central value and its uncertainty ctj X ^ ) 

as given in Table [TJ P(N; n) is the Poisson probability for observing N events 
where n = n s + rib ar e expected. p s (y§ r {i); a) and Pb(V~§ r (i)) are probability 
densities that the i-th event is due to signal or background, respectively. We 
calculate the likelihood as a function of A for a number of values of (M, T/M), 
where the value of M ranges from 400 to 800 GeV/c 2 at 50 GeV/c 2 intervals. 
T/M is considered at values of 0.05, 0.1, 0.2, 0.3, 0.4 and 0.5. 

The analysis method is tested with pseudo-experiments (PE's), where the 
background events are generated according to Table [l] and the total number of 
events is normalized to the 371 observed number of candidate events. An exam- 
ple of the analysis of a single pseudo-experiment is shown in Fig.0J Points repre- 
sent the observed VI distribution for a simulated signal sample with T/M = 0.1, 
M = 600 GeV/c 2 and A = 0.1. A is estimated by maximizing the likelihood L, 
given the observed \f§ r spectrum. The 95% C.L. lower/upper two-sided limit 
on A at the given mass and width is found by taking —2\a{L/L max ) = 3.84. 
We measure our expected sensitivity using a large number of null signal PE's. 



cm 90r 

o 




Figure 4: An example of a likelihood fit to \/*r spectrum for 1.9 fb 1 MC events (points), 
which includes the massive gluon with a T/M = 0.1, M = 600 GeV/c 2 , and A = 0.1. The 
solid line is the best fit to a superposition of g + G signal and the expected background (dash 
line) and SM prediction (dot-dash line). The likelihood fit is performed assuming F/M = 0.1 
and M = 600 GeV/c 2 . The inset shows the -21nL as a function of A. 
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We have studied contributions to the total uncertainty arising from system- 
atic effects. Variation of the tt invariant mass distribution affects the estimation 
of the coupling strength, A. The jet energy scale and top-quark mass are varied 
simultaneously (within their uncertainties) to properly account for their correla- 
tion. The uncertainties of the parton distribution functions (PDF) are estimated 
by comparing PE's where different PDF sets (CTEQ5L[22| vs MRST72[li|) are 
used; additionally, Aqcd and the CTEQ6M[22j PDF eigenvectors are varied. 
The systematic effect due to choice of MC generator is estimated by comparing 



PE's using events generated by pythia and herwig [24f . The scale of next- 
to-leading-ordcr systematic effects is estimated by using events generated with 
MC@NLO[25|. Systematic uncertainties in modeling initial- and final-state gluon 
radiation are estimated using pythia, where the range of QCD parameters are 
varied in accordance with CDF studies of Drell-Yan data [26J] . The uncertainty 
in the MC modeling of the multiple interaction and the 6-tagging efficiency as 
a function of jet pr are evaluated. All systematic uncertainties are evaluated at 
the full range of coupling strengths, gluon masses and decay widths considered, 
and are incorporated in the likelihood function. 



6. Results 

The tt invariant mass distribution observed in data is shown in Fig. [5j The 
best fit of A is consistent with the SM prediction, including a fluctuation of 
~ 1.7cr within the explored parameter range. We search for massive gluon 
with mass in the range [400,800] GeV/c 2 , though events with gluon masses 
beyond this region are included in the analysis. The 95% C.L. limits on the 
coupling strength A at T/M — 0.1 and T/M = 0.5 are shown in Fig. [6] A less 
stringent limit above 650 GeV/c 2 is due to 4 events on the high mass tail. The 
SM predicts 1.4 events above 850 GeV/c 2 , 4 events observed correspond to a 
Poisson probability of 5.4 %. The limits at 95% C.L. for several values of M 
and r /M are listed in Table [U The limits become weaker with higher M and 
wider T/M. 

In conclusion, we peform an exploratory search for a color-octet vector particle 
in general with minimal model dependence. No significant indication of the 
existence of massive gluon with |A| > 0.5 is observed in our search region of 
400 GeV/c 2 < M < 800 GeV/c 2 and 0.05 < T/M < 0.5. 
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KS=42.3% 

Data(L =1.9 fb 1 ) 

* int ' 

□ Top (175 GeV/c 2 ) 
I | EW & Single Top 
| | W+Light Flavor 
I I Non-W 
I I W+Charm 
I I W+Bottom 



500 600 700 800 900 1000 
tt invariant mass [GeV/c 2 ] 



Figure 5: The tt invariant mass distribution. The points represent the observed distribution; 
the histogram represents the SM prediction normalized to the observed data. The pink bands 
represent the uncertainty on the background estimations. 
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Figure 6: Excluded regions of A in the A-M plane (yellow), for Y/M = 0.1 (left) and T/M = 0.5 
(right). The expected limit is defined as median 95% value and ±1<t is denned with regard to 
the median for the set of null signal PE's. 
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Table 2: Expected and observed 95% C.L. lower/upper limits on A. Expected limits are in 
parentheses. 

r/M = o.or, r/M = o.io r/M = 0.20 r/M = 0.30 r/M = 0. 10 r/M = 0.50 

M = 400 (-0.036 / 0.042) (-0.046 / 0.086) (-0.074 / 0.156) ( -0.11 / 0.26) (-0.15 / 0.30) (-0.17 / 0.35) 

-0.043 / 0.040 -0.056 / 0.067 -0.089 / 0.11 -0.12 / 0.18 -0.16 / 0.20 -0.19 / 0.23 

M = 450 (-0.038 / 0.040) (-0.046 / 0.058) (-0.065 / 0.087) (-0.087 / 0.13) (-0.12 / 0.19) (-0.13 / 0.24) 

-0.045 / 0.027 -0.057 / 0.042 -0.086 / 0.06 -0.12 / 0.09 -0.15 / 0.14 -0.18 / 0.19 

M = 500 (-0.051 / 0.038) (-0.060 / 0.053) (-0.083 / 0.087) ( -0.12 / 0.13) (-0.13 / 0.18) (-0.16 / 0.24) 

-0.059 / 0.034 -0.067 / 0.043 -0.10 / 0.06 -0.14 / 0.10 -0.17 / 0.14 -0.20 / 0.19 

M = 550 (-0.058 / 0.049) (-0.075 / 0.069) ( -0.13 / 0.10 ) ( -0.15 / 0.15) (-0.20 / 0.20) (-0.31 / 0.23) 

-0.064 / 0.039 -0.083 / 0.055 -0.12 / 0.08 -0.16 / 0.13 -0.19 / 0.18 -0.23 / 0.20 

M = 600 (-0.074 / 0.058) ( -0.10 / 0.082) ( -0.19 / 0.12 ) ( -0.21 / 0.18) (-0.35 / 0.22) (-0.65 / 0.28) 

-0.073 / 0.048 -0.10 / 0.069 -0.16 / 0.10 -0.15 / 0.16 -0.22 / 0.20 -0.23 / 0.28 

M = 650 (-0.098 / 0.077) ( -0.14 / 0.092) ( -0.24 / 0.15 ) ( -0.36 / 0.20) (-0.58 / 0.27) (-0.80 / 0.33) 

-0.081 / 0.069 -0.096 / 0.083 -0.13 / 0.15 -0.16 / 0.20 -0.26 / 0.29 -0.34 / 0.35 

M = 700 ( -0.11 / 0.082) ( -0.16 / 0.12 ) ( -0.27 / 0.17 ) ( -0.44 / 0.25) (-0.68 / 0.32) (-0.90 / 0.42) 

-0.070 / 0.091 -0.068 / 0.14 -0.091 / 0.19 -0.13 / 0.29 -0.31 / 0.37 -0.37 / 0.47 
M = 750 ( -0.14 / 0.11 ) ( -0.20 / 0.14 ) ( -0.33 / 0.21 ) ( -0.50 / 0.31) (-0.71 / 0.39) (-1.00 / 0.52) 
-0.020 / 0.13 -0.021 / 0.18 -0.033 / 0.26 -0.03 / 0.38 -0-06 / 0.47 -0.37 / 0.61 
M = 800 ( -0.16 / 0.12 ) ( -0.24 / 0.17 ) ( -0.39 / 0.27 ) ( -0.59 / 0.37) (-0.80 / 0.49) (-1.09 / 0.62) 
-0.011 / 0.18 -0.017 / 0.26 0.017 / 0.37 -0.04 / 0.50 -0.01 / 0.63 -0.33 / 0.76 
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